Abstract: Due to extensive usage of Internet, substantial amount of data has extended widely over web, which serve access to particular data or to fetch more relevant data. It would be challenging to the search engine to provide quick results that is most relevant to the users. To search the relevant data and to reduce amount of time in fetching data, here propose the “Smart Crawler”. This returns most relevant data from the popular and most specific websites. It uses multiple search engines that processes the query provided by the user, cluster the results collected in a single platform and performs two stage crawling on data and URLs. In which in-site map generation is done to obtain relevant site with techniques such as reverse searching and page ranking.

Keywords: Deep Web, two stage crawler, ranking, in-site exploring, adaptive learning.